Circuit models have been developed to emulate an associative network of neurons as in a human brain. One such electronic implementation of a neuron is described in "A Neuromorphic VLSI Learning System", by Alspector and Allen, Advanced Research in VLSI, Proceedings of the 1987 Stanford Conference. In general, such systems provide an electronic representation of a neuron with many weighted inputs, a summing portion and a computational portion for determining a desired functional outcome given the weighted inputs. These neurons are coupled together to form an associative network or neural network. One such prior art neural network is shown in FIG. 1 of U.S. Pat. No. 5,055,897 titled "Semiconductor Cell for Neural Network and the Like", Ser. No. 225,035, filed Jul. 27, 1988 and assigned to the assignee of the present invention. Another related patent application is entitled "Method of Increasing the Accuracy of an Analog Neural Network and the Like", Ser. No. 634,033 filed Dec. 26, 1990, and assigned to the assignee of the present invention.
Training a network is necessary to make the network produce a desired output for a given input. In networks like a neural network, training the network includes programming and measuring analog levels of analog cells or floating gate devices of the network. These networks include EPROM (Electrically Programmable Read Only Memories) and EEPROM (Electrically Eraseable and Programmable Read Only Memories) devices that may use analog levels to increase storage density. In a neural network, training is accomplished by adjusting the weights for connections between nodes. Initially, all numeric weights for connections between neurons, as well as any weighting of input signals, are randomly set to various values. Signals are then input, and the output is observed. If an output signal is erroneous, then a mathematical computation will determine how the weights should be adjusted. Input signals are then re-applied and the output is again re-evaluated, until the output is correct. The technique for back-propagation or feedback of an error correction signal is an important characteristic in training a neural network.
Because training requires incremental adjustment of weighted connections in an electronic model of a neural network, floating gate devices are often used for the synapses or weighted connections. A synapse structure using a floating gate is described in reference to FIG. 2 of U.S. Pat. No. 4,802,103, Faggin et al. Another synapse cell utilizing a floating gate device is illustrated in FIGS. 2a and 2b of the co-pending application Ser. No. 225,035 cited above.
Floating gate devices are electrically chargeable devices typically used in an electronic synapse cell. A floating gate device can be used to represent an weight as an analog level of quantized charge stored on the floating gate. If a floating gate device is used for weighting in a synapse cell, the charge stored on the floating gate can be manipulated thereby affecting the conductivity of the synapse cell. In a neural network, because the level of charge on a floating gate is used to control current flow through a synapse cell, precise control over the level of charge stored on the floating gate is critical to the proper operation of the synapse cell and thus the neutral network. Prior art systems have experienced problems attributable to the inability to accurately set weights using floating gate devices. Floating gate manufacturing variations and physical changes in the devices after use have exacerbated the problems.
The level of charge and thus the weight stored on a floating gate device is set by applying a voltage pulse to the floating gate. The amplitude and duration of the voltage pulse must be carefully adjusted to achieve the proper weight on the floating gate. Prior art systems sometimes use processing logic or a software routine for setting weights in a neural network that uses floating gate devices. This prior art processing logic generally comprises a back propagation method using a binary search or other non-adaptive method for generating the proper voltage pulse (amplitude and duration). In a typical binary search method, an initial voltage pulse is generated and applied to the synapse cell. The charge on the gate is measured and compared to a desired weight to produce an error value. A new voltage pulse is then generated to reduce the error value from that of the prior iteration. In some prior art systems, the amplitude of the voltage pulse is halved or doubled in an attempt to reduce the error value. Since there is no linear relationship between the pulse amplitude and the floating gate charge, this prior art method is very inefficient.
The process of regenerating the voltage pulse continues until the measured weight is within a tolerance level of the desired or target weight. With the prior art binary search method, as many as 10 cycles may be needed to arrive at a correct weight. Also, because the voltage pulse applied to a floating gate device in any synapse call is indiscriminate, a binary search method treats each synapse cell of the network in the same or similar manner. Thus, even though adjacent synapse cells exhibit similar physical properties, the knowledge acquired by setting one synapse cell is not used in adjusting the weight of an adjacent synapse cell. Since there is no linear relationship between the pulse amplitude/duration and the floating gate charge, a linear based weight setting method will require more iterations to acquire a proper weight for each synapse cell processed.
A better method is needed for setting weights more precisely, quickly and adaptively in a neural network using floating gate devices. Greater precision allows a synapse cell to function more accurately in a neural network, and also reduces the overall training time. A reduction in the number of cycles necessary to achieve a correct weight lessens the computational requirements and enhances performance in adjusting the operation of a neural network. Also, an ability to adaptively set weighted values lessens the impact of physical changes in a floating gate device occurring over time.
Other prior art known to Applicant is "An Associative Memory Based On An Electronic Neural Network Architecture" by R. Howard et al., IEEE Transactions On Electronic Devices Vol. ED34 Jul. 1987; "An Artificial Neural Network Integrated Circuit Based On MNOS/CCD Principles" by J. Sage, Neural Network For Computing, AIP Conference Proceedings, 1986; "A 20V Four-Quadrant CMOS Analog Multiplier", by J. Babanezhad, IEEE Journal of Solid-Date Circuits, Vol. SC-20, Dec. 1985; "Programmable Analog Synapses For Microelectronic Neural Networks Using a Hybrid Digital-Analog Approach", by F. J. Mack et al., IEEE International Conference on Neural Networks, Jul. 24-27, 1988, San Diego, Calif.; "VLSI for Artificial Intelligence", edited by Jose G. DelGado-Frias and Will R. Moore, Kluwer Academic Publishers, pp. 230-33, 1989; "A Pipelined Associative Memory Implemented in VLSI", by Clark et al., IEEE Journal of Solid-State Circuits, Vol. 24, No. 1, pp. 28-34, Feb. 1989; and "A Neuromorphic VLSI Learning System", by Alspector and Allen, Advanced Research in VLSI, Proceedings of the 1987 Stanford conference.